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We consider the scheduling control problem for a family of uni- 
tary networks under heavy traffic, with general interarrival and ser- 
vice times, probabilistic routing and infinite horizon discounted linear 
holding cost. A natural nonanticipativity condition for admissibility 
of control policies is introduced. The condition is seen to hold for a 
broad class of problems. Using this formulation of admissible controls 
and a time-transformation technique, we establish that the infimum 
of the cost for the network control problem over all admissible se- 
quencing control policies is asymptotically bounded below by the 
value function of an associated diffusion control problem (the Brow- 
nian control problem). This result provides a useful bound on the 
best achievable performance for any admissible control policy for a 
wide class of networks. 

1. Introduction. In [12], Harrison introduced "controlled Brownian net- 
works" as formal approximations to stochastic network models under diffu- 
sion scaling. Since then, several authors have used such Brownian network 
models and the corresponding control problem [the so-called Brownian con- 
trol problem (BCP)] in obtaining control policies for the underlying queue- 
ing networks. In particular, there have been several works in recent years 
[1, 2, 5, 7, 20] that consider specific network models for which the corre- 
sponding BCP is explicitly solvable and, guided by the solution of the BCP, 
propose a control policy for the network which is then shown to be asymptot- 
ically optimal. However, there is a critical lack of mathematical theory which 
provides a rigorous basis for using the BCP as an approximating model for 
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a general family of controlled queueing networks. In this paper, we will con- 
sider a broad family of unitary networks (cf. [5]) under heavy traffic with 
general interarrival, service times and probabilistic routing with an infinite 
horizon discounted linear holding cost. We will prove that if a control policy 
is admissible (see Definition 2.6) for the network control problem, then the 
associated cost of using this policy is asymptotically bounded below by the 
value function of the corresponding diffusion control problem. This is the 
main result of the paper and is given in Theorem 3.1 (and in Corollary 3.2). 
Although the result is far from optimal — in particular one would also like 
to establish the reverse inequality — it is the first result, in the broad gen- 
erality we consider, that establishes any asymptotic relationship between 
the controlled stochastic network and its formal diffusion approximation. 
Indeed, even a proper formulation of admissible control policies for queue- 
ing networks that allows for weak convergence analysis to asymptotically 
relate it to the BCP has been critically lacking. Such a formulation, which 
in particular needs to include appropriate nonanticipativity conditions, is 
introduced in Definition 2.6. It will be seen in Theorem 5.4 that there exists 
a very large class of natural control policies which are admissible in the sense 
of Definition 2.6. (See also Proposition 2.8 for another sufficient condition 
for admissibility.) Roughly speaking, a control policy that does not change 
between two successive event times (an event is defined to be an exogenous 
arrival into the system or a completion of a service by some server), and 
at any event instant is a function of all the previous interarrival times, ser- 
vice times and routing decisions, is seen to satisfy the key requirements of 
admissibility in Definition 2.6. Once a proper formulation of admissibility 
is available, it becomes possible to use the machinery of weak convergence 
theory to relate the asymptotic value of the queuing control problem to the 
value function of the associated formal diffusion approximation. The main 
obstacle, in making rigorous the formal heavy traffic limit needed in the dif- 
fusion approximation approach, is proving the tightness of the control terms 
in the state evolution equation; see, for example, (2.22). Although for each r 
the control term Y r on the right-hand side of (2.22) is Lipschitz continuous, 
its Lipschitz constant increases to oo, linearly in r as r — > oo. One indication 
that the asymptotic analysis is somewhat delicate is that although the con- 
trol terms for the network control problem are continuous (in fact Lipschitz) 
for each fixed r, an optimal control in the limiting Brownian control prob- 
lem can in fact have jumps. In this paper, we develop a general approach 
to argue the convergence of the cost function which does not rely on the 
tightness of the various processes describing the dynamics, but rather on 
the tightness of suitable time-rescaled versions of the same. Although such 
time rescaling ideas go back to Meyer and Zheng [19] and Kurtz [15], their 
application in stochastic control problems was pioneered by Kushner and 
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coauthors [17, 18]. Indeed, inspiration for many ideas in the current paper 
comes from these latter works. In a related paper, Kurtz and Stockbridge 
[16] consider a control problem for a stable Jackson network with Markovian 
primitives. They are concerned with an ergodic cost problem associated with 
the approximating controlled diffusion; however, their techniques and goals 
are quite different from those of the current paper. 

The paper is organized as follows. The next section describes the queue- 
ing network and the associated control problem. Our key nonanticipativity 
condition on control policies is introduced in Definition 2.6. Two diffusion 
control problems, BCP and the corresponding equivalent workload formula- 
tion (EWF), which arise from formal diffusion approximations of the network 
control problem are introduced as well. These control problems are given a 
"weak formulation" which is natural in view of the weak convergence ideas 
underpinning all heavy traffic approximation methods. Section 3 contains 
the main result of this paper (Theorem 3.1). The proofs of all but one of the 
key results, namely Theorem 3.7, are contained in Section 3 as well. This 
latter theorem is at the heart of our analysis and its proof is given separately 
in Section 4. The main ingredients of the proof are the time transformation 
in Lemma 3.6 (which allows for appropriate tightness estimates) and re- 
lationships between various multiparameter nitrations, stopping times and 
martingales associated with the queuing model. A careful analysis of the lat- 
ter enables one to show that a typical limit point of the (time-transformed) 
control sequence has suitable adaptability properties, and the limit points 
of the "pre-Brownian motions" in the queuing model are martingales with 
respect to the desired filtration. We refer the reader to Remark 3.8 for more 
on these key adaptability issues. Finally, in Section 5 of this paper, we show 
that there exists a broad family of control policies which satisfy the admis- 
sibility requirements of Definition 2.6. 

The following notation will be used. The space of reals (nonnegative reals), 
positive integers (nonnegative integers) will be denoted by K N (No), 

respectively. For m > 1, C m will denote the space of continuous functions 
from [0, oo) to W 71 . For m > 1, T> m will denote the space of right continuous 
functions with left limits, from [0, oo) to W m with the usual Skorohod topol- 
ogy and B(T> m ) the corresponding Borel sigma-field. All continuous-time 
processes considered in this paper will have sample paths in T> m . All vec- 
tors will be column vectors and all vector inequalities are to be interpreted 
componentwise. We will call a function / £ T> m nonnegative if f(t) > for 
all t 6 M+ . A function / S T> rn is called nondecreasing if it is nondecreasing 
in each component. The weak convergence of processes Z n to Z as ele- 
ments of V m will be denoted by Z n ^~ Z . A sequence of processes {Z n } 
is tight if and only if the measures induced by the Z n 's on (T> m ,B(V m )) 
form a tight sequence. A sequence of processes with paths in T> m (m > 1) 
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is called C-tight if it is tight in D m and any weak limit point of the se- 
quence has paths in C m almost surely. For processes {Z n }, Z defined on 
a common probability space, we say that Z n converge to Z uniformly on 
compact time intervals (u.o.c.) in probability (almost surely) if for all t > 0, 
sup 0<s<t \Z n (s) — Z(s)\ converges to zero in probability a.s. The inherent 
nature of the problem makes this article heavy in notation. To ease the 
notational burden, standard notation (see, e.g., [6]) for different processes 
is used (e.g., Q for queue-length, I for idle time, W for workload process, 
etc.). We also use standard notation, for example, W,W, to denote fluid 
scaled, respectively, diffusion scaled, versions of various processes of inter- 
est. Time rescaling ideas used in the paper lead to processes [see (3.36)] that 
are obtained via time transformation of the original processes of interest; 
these are typically denoted with a superscript, as in W. Since we deal with 
vector-valued processes, we use both (a{,i = 1, . . . ,m;bi,i = 1, . . . ,n) and 
(ai, . . . , a m , b\, . . . , b n ) to denote a vector the first m components of which 
are given by a\, . . . ,a m and the next n components of which are given by 
&1, . . . , b n , and so on. The notation (c?, % = 1, . . . , m, j = 1, . . . ,n) corresponds 
to the vector (c\, . . . , c^, cf , . . . , c^, . . . , cj, . . . , c"). 

2. Stochastic network model and the control problem. Let (Q, J 7 , P) be 

a probability space. All the random variables associated with the network 
model described below are assumed to be defined on this probability space, 
unless specified otherwise. The expectation operation under P will be de- 
noted by E. 

Network structure. We begin by introducing the family of queuing net- 
work models that will be considered in this work. Figure 1 gives a schematic 
for such a model. The network has I infinite capacity buffers (to store I dif- 
ferent classes of jobs) and K nonidentical servers for processing jobs. Arrivals 
of jobs can be from outside the system and/or from the internal rerouting of 
jobs that have already been served by some server. Several different servers 
may process jobs from a particular buffer. Service from a given buffer i by a 
given server k is called an activity. It is assumed that there are J activities 
[at most one activity for a server-buffer pair (i,k), so that J < I • K]. The 
activities are labeled j = 1, . . . , J. It is also assumed that the integers I, J, K 
are strictly positive. 

The correspondence between activities and buffers, and activities and 
servers, are described by two matrices, C and A, respectively. C is an I x J 
matrix with Cij = 1 if the jth activity processes jobs from buffer i, and Cij = 
otherwise. A is a K x J matrix with A^j = 1 if the kth. server is associated 
with the jth activity, and A^j = otherwise. Each activity associates one 
buffer and one server, and so each column of C has exactly one 1 (and 
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similarly, each column of A has exactly one 1). We will further assume that 
each row of C (resp. A) has at least one 1, that is, each buffer is processed 
by (resp. server is processing) at least one activity. 

It is also assumed that once a job starts being processed by an activity, it 
must complete its service with that activity, even if its service is interrupted 
for some time (e.g., for preemption by a job from another buffer). When 
service of a partially completed job is resumed, it is resumed from the point 
of preemption — that is the job needs only the remaining service time from 
the server to be completed (preemptive-resume policy). Also, an activity 
must complete the service of any job that it has started before starting 
another job from the same buffer. An activity always selects the oldest job 
in the buffer that is not yet served when starting a new service [i.e., First 
In First Out (FIFO) within class]. 

Stochastic primitives. The networks considered in this paper will be 
nearly critically loaded, that is, "approaching heavy traffic," as made precise 
in Assumption 2.3 below. Mathematically, this is modeled by considering a 
sequence of networks {Af r }, each network in the sequence having identi- 
cal structure except for the rate parameters that may depend on r. Here, 
r£§C ]R + , where § is a countable set: {r\,V2, . . .} with 1 < r\ < T2 < • • • 
and r n — > oo, as n — > oo. For notational simplicity, throughout the paper, 
we will write the limit along the sequence r n as n — > oo simply as "r — ► oo." 
Also, r will always be taken to be an element of § and thus hereafter the 
qualifier rG§ will not be stated explicitly. The parameters of the networks 




Fig. 1. A unitary network. 
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are assumed to satisfy the heavy traffic condition (see Definition 2.2). The 
physical network that is "close to critical loading" corresponds to one of the 
networks in the sequence, for a large value of r. 

The rth network Af r is described as follows. If the ith class of jobs (i = 
1, . . . , I) has exogenous arrivals, the interarrival times are given by a sequence 
of nonnegative random variables {u\{n) : n > 1} that are i.i.d. with mean and 
standard deviation \ja\ and cr"' r £ (0, oo), respectively. Let, by relabeling 
if necessary, the buffers with exogenous arrivals correspond to i = 1, . . . , I', 
where I' < I. We set a[,cr"' r = and u[(n) = oo, n > 1, for i = I' + 1, . . . , I. 
Service times of the jth type of activity (for j = 1, . .. , J) are given by a 
sequence of nonnegative random variables {Vj(n) :n > 1} that are i.i.d. with 
mean and standard deviation and crj' r € (0, oo), respectively. We will 
assume that the above random variables are in fact strictly positive, that is, 

(2.1) For alH = l,...,I, j = l,...,J P«(l) > 0) = P(^ r (l) > 0) = 1. 

We will further impose the following uniform integrability condition: For all 
j = 1, . . . , J and all i = 1, . . . , I', 

(2.2) the sequences {{ul(l)) 2 } r and {(vj(l)) 2 } r are uniformly integrable. 

Rerouting of jobs completed by the jth activity is specified by a sequence 
of (1 + l)-dimensional vectors {(4>Q V (n), <^ J > r (n)), n > 1}, where <^' r (n) = 
(<^' r (n) : i = 1,. . . ,1). For each j and i = 0, 1, ... , I, ^' r (n) = 1 if the nth 
completed job by activity j gets rerouted to buffer i, and takes the value 
zero otherwise, where i = represents jobs leaving the system. It is assumed 
that for each fixed r, {(^>Q r (n), <^' r (n)), n > 1}, for j = 1,...,J are inde- 
pendent sequences of i.i.d. Multinomial (i+i)(l, (pbiP 3 )), where p 1 = (p? :i = 
1, . . . , J). That, in particular, means for j = 1, . . . , J, n > 1, J2i=o 4'i' r ( n ) = 
J2i=oPi = 1- Furthermore, for fixed j = 1, . . . , J, ii, i 2 = 1, . . . ,1, if cr^ 2 = 
Cov(^ r (n),^ r (n)), then 



(2.3) 



^(1-pj), ifii=i 2 = 



We also assume that for i = 1, . . . , I, j = 1, . . . , J, the sequences {u\(n) : n > 
1}, {i>J(n) :n > 1} and {<^' r (n),n > 1} are mutually independent. 

Next, we introduce the primitive stochastic processes (E r ,S r ) that de- 
scribe the state dynamics for the model. The process (E^, . . . ,Ep) is the 
I'-dimensional exogenous arrival process, that is, for each i = 1, . . . , I', EJ(t) 
is a renewal process which denotes the number of jobs that have arrived 
to buffer i from outside the system in [0,t]. For a class i to which there 
are no exogenous arrivals (i.e., i = I' + 1, . . . , I), we set E^(t) = for all 
t > 0. We will denote the process (£"[,..., Ej)' by E r . For each activity 



CONTROLLED STOCHASTIC NETWORKS 



7 



j = 1, . . . , J, there is a renewal process 5J such that Sj(t) denotes the num- 
ber of complete jobs that could be processed by activity j in [0,t] if the 
associated server worked continuously and exclusively on jobs from the as- 
sociated buffer in [0,t\. The vector (S[, . . . , is denoted by S r . More 
precisely, for i = 1, . . . , I, j = 1, . . . , J, mi > l,rij > 1, let 

mi Uj 

(2.4) er(mO = E «?(«)> ^KO = E «J(n). 

n=l n=l 

We set £[(0) = and rfAti) = 0. Then E%, 5J are renewal processes given as 
follows: 

El(t) = max{mi >0:^{rm) < t}, 

(2.5) 

= max{nj > 1 : r?J(nj) < t}, t> 0. 

Finally, we introduce the routing sequences. Let <l>^' r (n) denote the number 
of jobs that are routed to the zth buffer, out of the first n jobs completed 
by activity j. Thus, for i = 1, . . . , I, j = 1, . . . , J, 

n 

(2.6) *f(n)=E#V)» 71 = 1,2,.... 

771=1 

We will denote by {^' r (n)} the I-dimensional sequence {(<&{' r (ra), . . . , $j' r (n))'} 
corresponding to the routing of jobs completed by the jth activity. Also, the 
matrix ($> l,r (n), $ 2,r (?i), . . . , * J,r (n)) will be denoted by <& r (n). 

Control. A scheduling policy or control is specified by a nonnegative, 
nondecreasing J-dimensional process T r = {(T[ (t), . . . , TJ (i)), t > 0}. For 
any j = 1,...,J, t > 0, TJ(t) represents the cumulative amount of time 
spent on the jth activity up to time t. For a control T r to be admissi- 
ble, it is required to satisfy additional properties which are specified below 
in Definition 2.6. 

State processes. For a given scheduling policy T r , the state processes 
of the network are the associated J-dimensional queue length process Q r 
and the K-dimensional idle time process I r . For each t > 0, i = 1,...,I, 
Qi(t) represents the queue length at the ith buffer at time t (including the 
jobs that are in service at that time), and for k = 1,...,K, I£(i) is the 
total amount of time that the fcth server has idled, up to time t. Let q r = 
Q r (0) € be the J-dimensional vector of initial queue lengths. Note that, 
for j = 1, . . . , J, t > 0, Sj(Tj(t)) is the total number of services completed 
by the jth activity up to time t. Thus, the total number of completed jobs 
(by activity j) up to time t that get rerouted to buffer i = 1, . . . ,1 is given 
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by the process $>l' r (Sj (Tj (t))) . Hence, the state of the system at time t > 
is described by the following equations: 

Ql(t) = q r + El{t) - £ CySSpjit)) + £ *f (S?(2j(t))), 

, . i=i 3=1 

(2.7) 

i = l,...,I, 

j 

(2.8) F k (t) =t-J2 A kj TJ(t), k = 1, . . . , K. 

3=1 

We will only be concerned with admissible policies (see Definition 2.6) which, 
in particular, ensures that the queue-length and idle-time processes (Q r ,I r ) 
are nonnegative. 

Heavy traffic. We begin with a condition on convergence of various pa- 
rameters in the sequence of networks {A/" r }. 

Assumption 2.1. There are nonnegative q, a, a u , Q\ el 1 ;/?, a v , 62 € 
R 3 such that q > 0, a > 0, /? > 0, a u > 0, a" > (a*, of = if and only if 
i = I' + 1, . . . ,1), and 

0f = r (a r -a) ^ = r(/T - /?) -> 2 as r -> 00, 

(2.9) a u ' r ^a u , a v ' r -» a" asr^oo, 

..... 

<?= > g as r — > 00. 

r 

Next, we present the key heavy traffic condition, as introduced in [11] 
(also see [5, 6, 13]), on the sequence {M r }. 

Definition 2.2 {Heavy traffic). Define I x J matrices P' and R, such 
that P[j = p?, for i = 1, . . . , I, j = 1, ... ,3 and 

(2.10) R = {C -P')diag((3). 

We say that the sequence {Af r } approaches heavy traffic as r — > 00 if, in 
addition to Assumption 2.1, the following two conditions hold: 

(i) There is a unique optimal solution (x*,p*) to the following linear 
program (LP): 

(2.11) minimize p s.t. Rx = a, Ax<pe,x>0. 

(ii) The unique solution (x*,p*) of the above linear program satisfies 
p* = 1 and Ax* = e. 

Here e is a K-dimensional vector of ones, and for a vector a, diag(a) 
denotes the diagonal matrix such that the vector of its diagonal entries is a. 
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Assumption 2.3. The sequence of networks {M r } approaches heavy 
traffic as r — > oo. 

Assumption 2.3 will be a standing assumption for this paper and will not 
be mentioned explicitly in the statements of various results. 

Remark 2.4. From Assumption 2.3, x* given in (i) of Definition 2.2 is 
the unique J-dimensional nonnegative vector satisfying 

(2.12) Rx* = a, Ax* = e. 

Following [11], assume without loss of generality (by relabeling activities, if 
necessary), that the first B components of x* are strictly positive (corre- 
sponding activities are referred to as basic) and the rest are zero (nonbasic 
activities). For later use, we partition the following matrices and vectors in 
terms of basic and non-basic components: 

"TO" 

TZ(-) 



(2.13) 



x b 




T r {-) 



A=[B:N], 



where is a (J — B)-dimensional vector of zeros and B and N are K x B 
and K x (J — B) matrices, respectively. 

Other processes. The vector x* defined above gives the nominal alloca- 
tion rates for the J activities. Define the following deviation process as the 
difference between T r and the nominal allocation: 



(2.14) 



Y r (t)=x*t-T r (t), t>0. 



It follows from (2.8) and (2.12) that the idle-time process I r has the following 
representation: 

I r (t) = AY r (t), t>0. 

Next, we define a (K + J — B) x J matrix K and a (K + J — B)-dimensional 
process U r as follows: 



(2.15) K = 

where I above denotes a (J 



B 




N 



U r (t)=KY r (t), t>0, 
B) x (J — B) identity matrix. Note that 



(2.16) 



u r (t) 



F(t) 
T r n {t) 



t > 0. 



Another process that plays a crucial role in our analysis is the workload 
process. This is an L-dimensional process (L < I) defined as a suitable linear 
combination of the queue length processes as follows: 



(2.17) 



W r (t) = AQ r (t), 



t >0, 
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where A is an L x I-dimensional matrix, called the workload matrix, whose 
rows are determined by the optimal solution of the dual of the linear program 
in (2.11) (see [13]). We will work with one of the canonical choices of the 
workload matrix as defined in [13]. For this choice of A, one can obtain (see 
[6]) a nonnegative L x (K + J — B) matrix G such that 

(2.18) AR = GK. 

We make the following additional assumption on G: 

Assumption 2.5. Each column of G has at least one strictly positive 
entry: equivalently, there exists a c > such that for every u £ M.^ +J B ^ , 

\Gu\ > c\u\. 



The above will also be a standing assumption, so explicit reference to it 
will be omitted. 



Rescaled processes. The two scalings which are exploited in our analysis 
are the fluid scaling (corresponding to the law of large numbers) and diffusion 
scaling (corresponding to the central limit theorem). These are described as 
follows: 



Fluid scaled processes. These are obtained from the original process by 
accelerating time by a factor of r 2 and dividing space by r 2 . The following 
fluid scaled processes will be used: For t > 0, 

E r (t) = r~ 2 E r {r 2 t), S r (t) = r - 2 S r (r 2 t), 

(2.19) 3> j > r (t)=r- 2 ® j ' r ([r 2 t]), j = l,...,J, T r (t) = r" 2 T r (r 2 t), 

F(t) = r- 2 I(r 2 t), Q r (t) = r~ 2 Q(r 2 t). 

Diffusion scaled processes. These are obtained from the original process 
by accelerating time by a factor of r 2 and dividing space by r (after appro- 
priate centering). The following diffusion scaled processes will be considered: 
For t > 0, 

^ ^ (Sr(r 2 t)-pW 2 t) ^ 
r 

^.(EJWffl, j — o,i,..., j, 

(2.20) r 
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U r {t)=r- l U r {r 2 t), 
Q r {t)=r- l Q r {r 2 t), 

Y T {t)=r~ l Y T {r 2 t), 

where for x € M + , [x] denotes its integer part, that is, the largest integer 
bounded by x. Note that the last four processes do not need to be centered, 
as it will be seen that their fluid scaled versions converge to zero as r — > oo. 
Define, for t > 0, 

Xl{t) = EV(t) - - J*")$(2j(*)) 

(2.21) j 

+ ^4f(5J(f;(t))), i = l,...,L 

Recall the definitions of and g r introduced in Assumption 2.1. From the 
above definitions of rescaled processes, and equations (2.7), (2.8), (2.12), 
(2.15) and (2.17), one can establish the following relationships between the 
various diffusion-scaled quantities: For all t > 0, 

(2.22) Q r (t) =q r + X r (t) + [0[t - (C - P') diag(^')T r (t)] + RY r (t), 

(2.23) U r (t) =KY r (t), 

W r (t) = Aq r + AX r (t) + K[e\t - (C - P') diag(^)f r (t)] + ARY r (t) 

(2.24) 

= Aq r + AX r (t) + A[^« - (C - P') diag(^)T r (t)] + GL> r (t). 

In obtaining the above equations, we have also used (2.12). Finally, in ob- 
taining (2.24), we have used (2.18) and (2.23). 

Cost function. For the network N r , we consider an expected infinite 
horizon discounted (linear) holding cost associated with a scheduling policy 
T r , given in terms of the corresponding diffusion-scaled queue length process 
Q r as follows: 

/ poo \ / roc 

(2.25) J r (q r ,T r )=El^j e~^h ■ Q r (t) dtj +E(J e^p ■ dU r (t) 

where q r = Q r (0). Here, 7 6 (0, 00) is the "discount factor," h, an I x 1 vector 
with each component hi G (0, 00), i = 1, ... ,1, is the vector of "holding costs" 
for the I buffers and p > is a (K + J — B) x 1 vector of costs for incurring 
idleness in the system. 

We now introduce the key admissibility requirements on the control policy 
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Multiparameter filtrations and stopping-times. We refer the reader to 
Section 2.8 of [10] for basic definitions and properties of multiparameter ni- 
trations, stopping times and martingales. For m = (mi, . . . , mi) G N 1 , n = 
(ni, . . . , nj) G N J we define the multiparameter filtration generated by inter- 
arrival and service times and routing variables as 

f r ((m,n)) = a{u T i (m' i ),v r j (n' j ),(j)f r (n' j ): 

(2.26) 

m- <mi,n'j < nf,i= 1,...,I, j = 1,. .. , J}. 

Then {T r {{m,n)) ime^.neN- 1 } is a multiparameter filtration with the 
following (partial) ordering: 

(m ,n ) < (m 2 ,n 2 ) if and only if m\ < m 2 , < n 2 ; 

(2.27) 

i = j = l,...,J. 

Let 



(2.28) 




For all (m, n) € {0, 1} , we define J- r ((m, n)) = J- r {{\, 1)), where 1 denotes 
the vector of l's. 

Definition 2.6 [Admissibility of control policies). For a fixed r and 
g r G M 1 , a scheduling policy T r = {(T[(t), . . . , Tj" (t)) :t > 0} is called admis- 
sible for 7V r with initial condition g ,r if the following conditions hold: 

(i) TJ is nondecreasing, nonnegative and satisfies TJ(0) = for j = 
1,...,J. 

(ii) I£ defined by (2.8) is nondecreasing, nonnegative and satisfies 11(0) = 
for fc = l,...,K. 

(iii) Q[ defined in (2.7) is nonnegative for i = 1, . . . , I. 

(iv) Define for each r, t > 0, 

, x a J (t) = «' E (t)^ r o S (t)) 
(2 29) 

= (T[(r 2 t) • 1:/ 1 I: 5}(Tj(r 2 t)) + 1 : j = 1, . . . ,J). 

Then, for each t > 0, 

(2.30) cr5(i) is a {^((m, n)) :m G N 1 , n G N J } stopping time. 
Define the filtration {^{(t) :t > 0} as 

^(t) = r"K(t)) = a{A G : A n {a r (t) < (m,n)} G F((m,n)), 

for all m G N 1 , n G N J }, 

for all t > 0. Then 

(2.31) U r is {^i(t)} adapted. 
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A sequence of control policies {T r } is called admissible with initial condi- 
tion q if, for each r, T r is an admissible policy for the rth network with some 
initial condition q r and q r — > q as r — > oo. Denote the class of all admissible 
sequences {T r } with initial condition q by A(q). 

Remark 2.7. (i) and (ii) in Definition 2.6 imply, in view of (2.8) and 
properties of the A matrix, that 

(2.32) < Tj(t) - TJ(s) < t - s, j = l,..., J, for all < s < t < oo. 

In particular, TJ is a process with Lipschitz continuous paths. Condition (iv) 
in Definition 2.6 can be interpreted as a nonanticipativity condition. Indeed, 
in Section 5 of this paper we will show (Theorem 5.4) that a very large 
family of control policies that do not "anticipate the future" in a suitable 
sense satisfy the admissibility condition (iv) of Definition 2.6. 

The following proposition gives another sufficient condition for part (iv) 
of the above definition to hold: 

Proposition 2.8. Let r and q r G M 1 be fixed and suppose that T r is a 
scheduling policy that, in addition to (i)-(iii) of Definition 2.6, satisfies, for 
all t,tj G [0,oo), j = 1, J and m G N 1 , n G N J , 

{Tj(t) < tj,j = 1, . . . , J} n {El{t) =rm,i = 1, ■ •• ,1} 

(2.33) 

n {STjitj) = n 3 ,j = 1, . . . , J} G T r {{m,n)). 
Then T r is admissible for J\f r with initial condition q r . 

Proof. We first prove (2.30). For this, it suffices to show that for any 
m G N 1 , n £ N J , 

(2.34) {a r f(t) < m l ,a r j(t) < n 3 , i = 1, . . . , I, j = 1, . . . , J} G r((m,n)). 
The set above can be written as 

X; <K)>A J2 v r j (n' j )>TJ(r 2 t),i = l,...,I,j = l,...,j\ 

lm'=l n'-=\ > 

= U {Tj(r 2 t)< qj ,j = l,...,3} 

n I J2 u r i ijn' i )>r\ ^ ^(n^) > = 1, . . . ,1, j = 1, . . . , J \ 

lm<=l n'.=l J 

(2.35) 
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U -AgnBg 



= |J |J A q nB cl n{E r (r 2 t)=m,S r (q)=h}, 

g€Q J (rh,h)<(m,n) 

where Q denotes the set of rational numbers. From (2.33), it follows that 
for each q and (rh,h) < (m,n) the set A q H {E r (r 2 t) = m,S r (q) = n} € 
^((m, n)) C F '((m,n)). Also, clearly, B q G J- r ((m,n)) for every q G Q J . 
Thus, we have that the set in (2.35) is in .F r ((m,n)) and hence (2.34) fol- 
lows. 

Since 0"g(i) is a nondecreasing function of t, it follows that {J~x(t) :t > 0} 
is a filtration. In order to prove (2.31), it suffices to show that TJ(r 2 -) is 
adapted to the filtration {T{(t) :t > 0}. For this, it is enough to show that 
for fixed t > and real ay, j = 1, . . . , J, we have {TJ(r 2 t) < a,j , j = 1, . . . , J} € 
^f(t). The last statement will follow if the following property holds for all 
m, n: 

{Tj(r 2 t)<a i ,j = l,...,J} 

n {°"o',f (*) ^ m *> °"o'j (*) < = 1, . . . ,1, j = 1, . . . , J} G ^"((m, n)). 
The set above can be written as 



|J {TV(rH)< qj J = l,...,3} 



n\ J2 <K)>r 2 f, ^ uJ(n^)Aaj-J >q j ,i = l,...,I,j = l,...,3\. 



~m =1 \n =1 



Finally, proceeding exactly as in the proof of the first part, one obtains that 
this set is in ^((m, re)). This completes the proof of the proposition. □ 

Given a sequence of admissible control policies {T r }, with initial condition 
sequence {q r } such that q r — ► q as r — ► oo, the associated asymptotic cost is 
defined as 

(2.36) J(g J {T r })=liminf J r (q r X)- 

r — >oo 

Let the infimum of the asymptotic cost over all admissible sequences {T r } 
be denoted by J*(q), that is, 

(2.37) J*(q)= inf J(q,{T r })- 

{T-}€A(q) 

The main goal of the paper is to give a lower bound for this infimum in 
terms of the value function of an associated diffusion control problem. 



CONTROLLED STOCHASTIC NETWORKS 



15 



Associated diffusion control problems. We now introduce the formal dif- 
fusion approximation to the network control problem described above. It will 
be seen in Lemma 3.3 [part (C)], via an application of a functional central 
limit theorem, that X r defined in (2.21) converges weakly to a Brownian 
motion X with drift and covariance matrix 

J 

(2.38) S = X" + (C - P')E V diag(x*)(C - P')' + ^[E^-x*], 

i=i 

where S n is an I x I diagonal matrix with diagonal entries (af ) 2 , i = 1, . . . , I, 
E" is a J x J diagonal matrix with diagonal entries (o~j) 2 , j = 1, ... ,3 and 

the S^'s are I x I matrices with entries c^jii,^ = 1, ... ,1 [cf. (2.3)]. Fur- 
thermore, it will be seen in Lemma 3.3 [part (B)] that for all "reasonable" 
control policies, the third term on the right-hand side of (2.22) converges to 
9t, where 

(2.39) e = e 1 -(C-P')diag(9 2 )x*. 

Although the weak convergence of the last term (indeed, even its tightness) 
in the right-hand side of (2.22) is far from obvious on formal passing to the 
limit, one is led to the following diffusion control problem: 

Definition 2.9 (Brownian control problem (BCP)). A J-dimensional 
adapted process Y, defined on some filtered probability space (O, !F, P, {^(t)}) 
which supports an I-dimensional {.F(t)}-Brownian motion X with drift 
and covariance matrix £ defined in (2.38), is called an admissible control for 
the Brownian control problem with the initial condition q iff the following 
two properties hold P-a.s.: 

Q(t) =q + 0t + X(t) + RY(t) > for all t > 0, 
U(t) = KY(t) is nondecreasing and U (0) > 0, 

where 6 is as in (2.39). We refer to T = (Cl,J-,¥, {.F(t)},X) as a system and 
X as a (0,S) -Brownian motion. We denote the class of all such admissible 
controls by A(q). The Brownian control problem is to infimize 

/•OO P 

(2.40) J(Y) = E e~" ft h ■ Q(t) dt + E e^p ■ dU{t) 

JO J[0,oo) 

over all admissible controls Y G A(q) . Define the value function 



(2.41) 



J*(q)= inf J(Y). 
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Since an admissible control is not required to be of bounded variation, the 
above control problem is somewhat nonstandard and is difficult to analyze 
directly. However, as shown in [11], under suitable conditions, one can replace 
this control problem by an equivalent, more classical control problem which 
is also typically of a much lower state dimension. This control problem, 
in stochastic control terminology, is a problem of singular control with state 
constraints (SCSC) and is given as follows. [In the literature, this problem is 
also referred to as the equivalent workload formulation (EWF) of the BCP.] 

Effective cost function. The definition of the equivalent workload formu- 
lation (EWF) is tied to the notion of an effective cost function, described as 
follows. Recall the definition of the workload matrix A introduced in (2.17). 
Let W = {Az : z G For each miEW, define 

(2.42) h(w)=mm{h-q:Aq = w,q>0}. 

It is well known (see Theorem 2 of [4]) that one can take a continuous 
selection of the minimizer in the above linear program. That is, there is a 
continuous map q* : W — > such that 

(2.43) Q*(w) G argmin{/i • q : Aq = w, q > 0}. 

As an immediate consequence of these definitions, if w(q) = Aq, q G and 
w r ,w G W, then 

h-q>h(w(q)) for all gGl^, 

(2.44) 

h(w r ) — ► h(w) whenever w r ->ioasr-> oo. 

The function h(-) will be referred to as the effective cost function. The equiv- 
alent workload formulation (EWF) and the associated control problem is 
defined below. Let 

(2.45) lC = {ueR K+3 ~ B \u = Ky,y£R J }. 

Definition 2.10 (Equivalent workload formulation (EWF)). A (K + 
J — B) -dimensional adapted process Uq, defined on some filtered probability 
space (Cl,jF, P, {^F(t)}) which supports an I-dimensional { T(t) }-Brownian 
motion X with drift and covariance matrix E defined in (2.38), is called an 
admissible control for the equivalent workload formulation of the Brownian 
control problem with the initial condition q iff the following two properties 
hold P-a.s.: 

Uq is nondecr easing, Uq(0) > 0, Uo(t) G /C for all t > 0, 

(2.46) 

W(t) = w + A9t + AX(t) + GU (t) G W for all t > 0, 
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where is as in (2.39). We denote the class of all such admissible controls 
by Ao(w). The control problem for the equivalent workload formulation is 
to infimize 

(2.47) J (U ) = E e-^h(W(t))dt + E e'^p ■ dU (t) 

JO J[0,oo) 

over all admissible controls Uq 6 A${w). Define the value function 

(2.48) J » = . inf J (U ). 

U £A (w) 

From Theorem 2 of [11], it follows that for all w € W, q € satisfying 
w = Aq, 

(2.49) J*(q) = 4(w). 

3. Main result and proofs. In this section, we present the main theorem 
of this paper (Theorem 3.1) and provide proofs of the key results that are 
used in proving the theorem. 

Theorem 3.1. Let {T r } be an admissible sequence of control policies 
for the initial condition q, that is, for each r , T r is an admissible policy for 
M r with some initial condition q r , and q r — > q as r — > oo. Let J(q, {T r }) = 
liminf r _ +00 J r (q r , T r ) be the associated asymptotic cost. Let w = Aq and 
Jq(w) be, as in (2.48), the value function in the EWF. Then 

(3.1) J(q,{T r })>J^w). 

As a corollary of Theorem 3.1 and Theorem 2 of [11] [see (2.49)], the inn- 
mum of the cost in the network control problem is asymptotically bounded 
below by the value function of the BCP: 

Corollary 3.2. Fix q e R\ and set w = Aq. Let J*(q), J*{q) and 
Jq{w) be given via (2.37), (2.41) and (2.48), respectively. Then 

(3.2) J*(q)> J » = J*(q). 

The rest of the paper is devoted to the proof of Theorem 3.1. Thus, 
henceforth we will fix an admissible sequence {T r } € A(q) such that T r 
is an admissible policy for the initial condition q r and q r — > q as r — > oo. 
Furthermore, we will assume without loss of generality that 



(3.3) 



J(g,{r r })=liminf J r (q r ,T r ) < oo, 
r — *-oo 
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since otherwise inequality (3.1) holds trivially. Finally, the subsequence {r'} C 
{r} along which the above liminf is attained will be relabeled once more as 
{r}. In particular, by this relabeling, 

(3.4) J(q,{T r })= lim J r (q r ,T r ). 

r — *oo 

We begin with some preliminary results. 



Lemma 3.3. (A) The following functional central limit theorem holds: 
(3.5) 



(E r ,S r ,¥' J ,j = l,...,3) 



=> (E,S,& ,j = l,...,J) as r — > oo, 

where E,S,& J , j = 1, . . . ,3 are independent Brownian motions with drift 
zero and covariances 

£u j = i, . . . ; J [ c f (2.38)], respectively. 

(B ) The following functional law of large numbers holds: 



(Q r ,r,T r ,E r ,S r ,&> r , j = l,..., 3) 

(3.6) 

=>■ (0, 0, T, ae, fie,p?e, j = 1, . . . , J) as r — > oo, 

where denotes the process that equals a.s. at all times, e(t) = t, T*(t) = 
x*t,t > and x* is as in Remark 2.4. 
(C) Let X r be as in (2.21) and define 

(3.7) x 3 {.) = Ej(.) - Y,(Cij - r^Sjir,) ■ X^i.'-;; i r ). 

j j 
Then X is a (0, E) -Brownian motion and 

Z r = {EH-), S r j(TJ(-))> M ,r (^(Tj(-))),^r (0 -i 1 I,/ 1 1! 

(3.8) 

^(E i (-),S j (x* r ),^i(xp r )X i (-):i = l,...,I,j = l,...,3) = E. 

Proof. Weak convergence of E r ,S r to Brownian motions follows from 
the functional central limit theorem for renewal processes (see Theorem 

14.6 of [3]), and the weak convergence of $ to a Brownian motion with 
the stated covariance matrix follows from Donsker's theorem (Theorem 8.2 
of [3]). Finally, the joint weak convergence in (3.5) and claimed mutual in- 
dependence follow from the mutual independence of the primitive processes 
on the left-hand side of (3.5). This proves part (A). 

Part (C) is an immediate consequence of (A) and (B) on using Lemma 
3.14.1 of [3]. Finally, for the proof of (B), note that from (3.5) it follows that 

(E r ,S r ,&> r , j = l,..., 3) 

(3.9) 

(ae^e^e, j = l,...,J) as r — > oo. 



CONTROLLED STOCHASTIC NETWORKS 



19 



Also, from Definition 2.6, we have that for each r, T r is Lipschitz continuous 
with Lipschitz constant less than or equal to 1, and that T' r (t) < t for all 
t > (see Remark 2.7). Thus the same properties hold for the fluid-scaled 
processes T r for each r. This shows that the sequence {T r } is C-tight. In 
addition, we have from (2.7), (2.8) and the definition of fluid-scaled processes 
in (2.19) that 

(3.10) Ql(t) = f 2 + E\{t) - £) QjS^fJit)) + £ *f (£j(Tj(i))), 

3=1 3=1 

(3.11) r k {t)=t-Y,A k fi{t), 

3=1 

for all i = 1, . . . , I, fc = 1, . . . , K. Combining the above representations with 
(3.9) and the C-tightness of {T r }, we see that 

(3.12) {(Q r ,I r ,f r ,E r ,S r ,&' r , j = l,...,J)} is C-tight. 

Hence, it is enough to prove that any weak limit point of the sequence in 
(3.12) is given by the right-hand side of (3.6). Consider a further subse- 
quence (we use the same subscript r to denote this subsubsequence) which 
converges weakly to some (Q,I,T,E,S,*& J , j = 0,1, ... ,3). Using the Sko- 
rohod representation theorem, we can assume without loss of generality that 

(Q r ,I r ,T r ,E r ,S r ,&> r , j = l,... ,3) 

(3.13) 

— > (Q,I,T,E,S,<$> 3 , j = 1, ... ,3) u.o.c. as r — >■ oo, 

with probability 1. Since we have J(q,{T r }) < oo, using Fatou's lemma we 
get 

J r {q r ,T r ) 







(3.14) 



lim 

r— >oo 

> lim inf E 

r— »oo 



> E 



E 



poo 

/ e-^h-Q r {t)dt 
Jo 

e~ 7i (fiminf h ■ Q r {t)^j dt 



- Jt h-Q{t)dt 



Recalling that h > 0, we have, in view of the path continuity of Q, that 
Q = 0, a.s. Hence, from (3.9), (3.10), (3.11), (3.12) and the definition of R 
in (2.10), we obtain that 



(3.15) 







ae 



RT, 



< I = e - AT. 
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Finally, combining (3.15) with Assumption 2.3 [see Definition 2.2(ii) and 
Remark 2.4] yields 

(3.16) f(t) = x*t, I(t) = et-Ax*t = 0, t>0. 

This completes the proof of part (B). □ 

The following corollary is a direct consequence of Lemma 3.3: 



Corollary 3.4. Let 
(3.17) 



AdT (■) = (Mf > r (-) : i = 1, . . . , I, M/' r (-) : j = 1,...,J, 

Mf^(.):i = l,...,I) 



where for i = 1, . . . ,1, j = 1, . . . , J and i > 0, 

Nr?>'(t)±%(t), M?> r (t) = Sr(TJ(t)), 



(3.18) 



Then 



(3.19) 



M^(t) = ^df(5;(77(t))). 



M r (-) M'(-)= U(-),i = l,...,I,^K-), J = !,-••, J, 



£*?(/5^-), • = ! I], 

i=i / 

where E,S and are as in Lemma 3.3(A). 

Lemma 3.5. There exists a c* € (0,oo) swc/i i/iai /or a// i = 1, . . . , I, 
j = l,...,J, r > 1 and t>0, 



E 



sup |i£( S )|' 

0<s<t 



(3.20) 



E 



sup |fij(27( a ))|^ 
l_0<s<t 



E 



sup |*f(^(Tf( S ) 



I0<s<t 

Furthermore, for m = 1, . . . , K + J — B, t > 0, 
(3.21) hmsupE[L> (*)] < oo. 



<c*(t + l) 
<c*(i + l) 
<c*(t + l) 
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Proof. We first consider the third bound in (3.20). Fix i E {1, . . . , I}, j G 
{1,...,J}. Note that 



sup |*f(SJ(T/( S )))| < sup |*r(5;( S ))| 



(3.22) 



0<s<t 



0<s<t 

2 1 
<- + - 



sup irK ,s (^))i 

0<s<t 



where (f (n) = ELi(# r (*0 - p?), n = 1,2,... and o£ s (a) = SJ(r 2 S ) + 1, 
s > 0. The above bound follows from the fact that TJ(s) < s, s > 0, and that 
\(/){' r {k) -p{\< 2, for all k = 1,2,.... Define 

jT( n ) = {^>(fc),^:(A;) : A;<n}, n > 1. 

It is easy to check that {C/' r ( n )} is a martingale and a r j S (s), for each s > 0, 
is a stopping time with respect to the filtration {^(n)}. Furthermore, the 
quadratic variation of this martingale is given as 



(3.23) <C? ,r >(«) = n = l,2,.... 
Next, note that for all n = 1, 2, . . . , 

(3.24) EflCr (n + 1) - Cr (n) | |^(n)] = E|# r (n + 1) - p?'| < 2, 



and that {a r ^ S (s):s > 0} is a nondecreasing sequence of stopping times, 
satisfying 



E[af (s)]=E(S r J (rh) + l) 



(3.25) 



< ci(r 2 s + 1) < oo, s>0, 



where c\ € (0, oo) is a constant independent of r and s. This inequality is 
a consequence of Assumption 2.1 and well-known moment inequalities for 
renewal processes. Thus, using an optional sampling theorem (see Theo- 



rems 4.7.4, 4.7.5 of [9]) we obtain that {Cf'Vy ( S ))>^K ( s )) :s > °l is a 



;S ( \\ <w r,S i 

(.s)),H°j 

martingale. Using (3.23) and (3.25), we have, for all t > 0, 

nci' r (^ s (tm=^E\af{t)] 



(3.26) 



Taking the expectation of the squares of both sides of (3.22) and using 
Doob's inequality, we obtain that 



E 



sup |*f (S^f^s) 

0<s<t 
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< 2 



S up icrK ,5 (*))P 

T \0<s<t 



<£-(l+E|Cr(af(t))r) 



Combining the above display with (3.26), we obtain the third bound in 
(3.20). 

We will next show that for some c* G (0, oo), we have for all j = 1, . . . , J, 
r > 1 and t > that 



(3.28) 



E 



sup |ST(s) 

0<s<< 



<c*(t + l). 



Proof for the first term in (3.20) is almost identical to the proof of (3.28), and 
the second bound in (3.20) follows from (3.28) using the fact that 2J (t) < t 
for all t > 0. Define for t > 0, n = 1, 2, . . . 

1 n 

(3.29) ^ = -£(1-/3;^)), ef' r (t)= sup - fj^af (s))\. 



k=l 



0<s<t 



As a convention, we set fjj(0) = 0. A straightforward calculation shows that 

1 



£ f(i)<^-[sup v](af(s)) 



+ 



(3.30) 



< 



sup |^(SJ(r^) + l)-^(r^))| 

0<s<t 



+ 



< 2 



sup |^(^(rV) + l) 

0<s<t 



+ 



Hence, from (3.29) we have that 
(3.31) 



sup \S^s)\< sup \^(af(s))\+ef r (t) 

0<s<t 0<s<t 



<3 



sup |^(o"i' S (*))l 



+ 



Note that (n)} is a square integrable {^-"(n)}-martingale and for that each 



s > 0, er-' (s) is a {JP"(n)}-stopping time. Hence, an argument analogous to 
the one leading to (3.27) shows that 

E 



sup |^r( s ))r <4E[|^(aJ' a (*))H 

0<s<* J 



(3.32) 



<4 



^ J E[ffJ(r 2 t) + l] 
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where, once again, C2 is a constant independent of r and t. Combining this 
estimate with (3.31) proves (3.20). 

Finally, we consider (3.21). Since h is strictly positive, one can find C3 € 
(0, 00) such that 

\W r (t)\ = \AQ r (t)\ < \A\\Q r (t)\ < c 3 h ■ Q r (t), r > 1 and t > 0. 
Thus, from (3.3) we get 

/•oo 

(3.33) limsupE/ e~ ls \W r (s)\ds < 00. 

r— >oo JO 

Furthermore, from (2.24), (3.20) and Assumption 2.5, one can find C4 6 
(0, 00) such that 

cW r m (t) < cK\U r (t)\ < E\GU r (t)\ < E\W r (t)\ + c 4 (t + 1), 

for all m = 1, . . . , K + J — B, r > 1 and t > 0. Combining this with (3.33), 
we have 

/•OO 

(3.34) limsupE / e^U^s) ds < 00. 

r— >oo JO 

Finally, using the monotonicity of U^, we get, for all t > 0, 

/•oo /"t+l 

E/ e-^ s U^(s)ds>E e-^U r m {s)ds>e-^ t+l) E{U r m (t)). 

JO Jt 
Combining this inequality with (3.34), we have (3.21). □ 

In the following lemma, we introduce the time transformation which plays 
a crucial role in the proof of Theorem 3.1: 

Lemma 3.6. For each r, define the process r r (-) with values in [0, 00) 
as follows: 

K+J-B 

(3.35) r r (t)=t+ U r m (t), t>0. 

m=l 

Then the map t t : [0, 00) — ► [0, 00) is (almost surely) continuous and strictly 
increasing. Define f r (t) = inf{s > : r r (s) > t}, t>0. Consider the following 
time- transformed processes: 

W r {-) = W r (f r (-)), X r (-) = X r (f r (-)), 

(3.36) 

U r (.) = U r (f r (-)), M T {-) = M r (f r (-)). 

Then 

(A) {(W r ,X r ,U r ,f r ,E r ,M r ,M r } r is C -tight. 
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(B) Let a weak limit point of the above sequence, (W(-),X(-), U(-),f (-),H(-), 
M'(-),M(-)), be given on some probability space (Cl,J-,¥). Then, almost 
surely, 

(3.37) f (t) — > oo as t—> oo 

Define r(t) = inf{s > : f(s) > t}, t> 0. With probability one, f is continu- 
ous, t is right continuous and both r, f are nondecreasing maps from [0, oo) 
to itself. 

(C) The process M' equals the right-hand side of (3.19), with E,S and 
3> as in Lemma 3.3(A). Furthermore, the following equality holds almost 
surely: 

M{-) = {M E {-),M S {-),M^(-)) 

(3.38) = ^(f(-)), i = !,...,!, ^(**f(.)), j 

J 

i=i 

Proof. From Remark 2.7, we have that < TJ(t) - Tj(s) < t - s for all 
< s < t and j = 1, . . . , J. This, along with (2.14) and (2.23), shows that r r 
has Lipschitz continuous paths. Also, since is nonnegative, r r is strictly 
increasing. This proves the first statement in the lemma. Also, part (C) is a 
direct consequence of Corollary 3.4 and parts (A) and (B). 

Finally, we consider parts (A) and (B) of the lemma. Fix m G {1, . . . , K + 
J — B}. From (3.35), we have, on recalling that L^(-) is nondecreasing and 
nonnegative, that for < s < t and all r, 

r r (t) - r r (s) > (t - s) > 0, r r (t) - r r (s) > U r m (t) - U r m (s) > 0, 

and so 

(3.39) 0<f r (t)-f r (s)<(t-s), 0<U r m (t)-U^(s)<(t-s). 

This proves that f r {-) and U r (-) have continuous sample paths and that 
{f r (-)}r and {U r (-)} r are C-tight. The C-tightness of {X r (-)} r and {W r (-)} r 
is now immediate from Lemma 3.3(C), on noting that X r (-) = X r (f r (-)) and 
W r ( • ) = Aq r + A6f r ( • ) + AX r (■) + GU r (■). Also, the C-tightness of {M r , M r } 
follows from Corollary 3.4. This proves part (A) of the lemma. 

For the proof of part (B) of the lemma, note that the continuity and non- 
decreasing property of f are immediate consequences of the same properties 
of f r , for each fixed r. It now suffices to show (3.37), since the remaining 
properties are then immediate. Observing that f r is nondecreasing and f r 
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f as r — > oo, it follows that in order to prove (3.37), it suffices to show for 
all M > 0, 



lim limsupP[f r (t) < M) = 0. 

t >-00 t> — |,(~x~] 



(3.40) 

Using Markov's inequality, we see that 



\f r {t)<M]=F[T r (M)>t] 

(K+J-B) 



(K+J-B) 

M+ J2 U r m (M)>t 

m=l 



(3.41) 



U r m (M) > 



< E 

m=l 
(K+J-B) 

< £ (k + j-b; 

m=l 



(i-M) 



(K + J-B) 

E[tfc(M)] 
(t-M) • 



The statement in (3.40) now follows, using (3.21) and passing to the limit 
in (3.41), first as r — > oo and then as t — > oo. □ 

The following theorem is at the heart of our analysis. Since the proof is 
rather long, we postpone it to Section 4. 

Theorem 3.7. Let (W,X,U,t,t,E,M',M), given on the probability 
space (Q,J-,1P), be as in Lemma 3.6. Define 



(3.42) 



W(-) = W(r(-)), X(-) = X(t(-)), 
Z7(.) = U(r(-)), M(-) = M(t(.)). 
Then, almost surely, M = M' , that is, 

M(.) = (M E (-),M S (-),M*(-)) 

(3.43) = \ E{-),i = l,...,l,S j {x*-),j = 1,..., J, 

Y,^Uf3 j x*-),i = l,...,l). 

j=l ) 

Furthermore, there is a filtration {J-(t)}t>o on the probability space (Cl,j-,lP) 
such that the processes (W,X,U) are adapted to {fi(t)} and X is a {F(t)}- 
Brownian motion with zero drift and covariance matrix S given by (2.38). 
That is, F = (0,,^F, P, {F(t)},X) is a system in the sense of Definition 2.9. 
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Remark 3.8. From the relation X r (-) =X r (f r (-)), Lemma 3.3(C) and 
Corollary 3.4, it follows that X(-) = X(f (•)), a.s., where X is defined via 
(3.7). Combining this with the fact that f(r(t)) = t, t > 0, we obtain that 
X(-) = X(t(-)) = X(-). This immediately shows that X is a (0, X)-Brownian 
motion and hence a martingale with respect to its own filtration. However, 
it is not clear that the limiting control process U(-) will be adapted to this 
filtration (i.e., the filtration generated by {X(t)}). Theorem 3.7 shows that 
there is a filtration to which all the limit processes are adapted, and with 
respect to which the limit X is still a Brownian motion. 

Corollary 3.9. Let T be the system obtained in Theorem 3.7. Define 
w = Aq. Let (W, U) be as in (3.42). Then U € Aq(w) and (2.46) in Definition 
2.10 holds with (W,X, Uq) replaced by (W, X, U). In particular, we have 

(3.44) E / e-^h(W(t))dt + E / e" 7 *p ■ dU(t) > J * (w), 

JO J[0,oo) 

where E denotes the expectation operator corresponding to P. 

Proof. Since U and r are both nondecreasing, it follows that [/(•) = 
U (r(-)) is nondecreasing. Also, since U r (t) € /C, for all r and t > 0, we obtain 
that U(t) G K, t > 0. Using (2.24) and (3.36), we have that for all t > 0, 

W r (t) = Aq r + A[6{f r (t) - (C - P') diag(^)f r {f{t))\ 

(3-45) 

+ AX r (t) + GU r {t)eW. 
Taking the limit as r —> oo, one obtains that for all t > 0, 

(3.46) W(t) = w + A0f (t) + AX(t) + GU{t) £ W. 

Thus, from (3.42), and recalling f(r(£)) =t, t > 0, we have for all t > 0, 

(3.47) W(t) =w + A9t + AX{t) + GU{t) e W. 

From Theorem 3.7, we have that X(-) is a {^ r (t)}-Brownian motion and 
£/(•), W(-) are {^"(£)}-adapted. Hence, U G^4o(^)) with associated system 
r. The inequality in (3.44) is now an immediate consequence of the definition 
of the value function of the EWF, Jq(w), in (2.48). □ 

For the proof of the following lemma, we refer the reader to Theorem IV. 4. 5 
of [21]. 

Lemma 3.10. Let a be aM + -valued, right- continuous function on [0, oo) 
such that a(0) = 0. Let c be its right inverse, that is, c{t) = inf{s > : a(s) > 
t}, t>0. Assume that c(t) < oo for all t>0. Let f be a nonnegative Borel 
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measurable function on [0, oo). Then ifG is any K+ -valued, right- continuous 
function on [0, oo), 

(3.48) / f(s)dG(a(s))= [ f(c(s-))dG(s), 

J[0,oo) J[0,oo) 

with the convention that the contribution to the integrals above at is /(0)G(0). 
This, in particular, implies 



(3.49) / f(s)da(s) = f(c(s))ds. 

J[0,oo) J[0,oo) 

The following lemma will be needed in our proofs. We refer the reader to 
[8] (Lemma 2.4) for a proof. Let C d denote the space of all bounded functions 
in C d . 

Lemma 3.11. Suppose £ r — > £ in T> d and X r — > A inC as r — > oo. Further, 
suppose that X r is nonnegative and nondecreasing for each r. Then for any 
f sCj = space of bounded functions in C d , 



f(e(t)) d\ r (t) - / /(£(*)) dX(t) asr^oc, 
[0,u) J[0,u) 

uniformly for all u in any compact subset of [0, oo) . 

Proof of Theorem 3.1. On noting that f r {-) is continuous and f r (t) | 
oo as t — > oo, we have that 

/*oo / /*oo 

r{q r , T r ) = E / e^h ■ Q r {t) dt + E[ / e~^p ■ dU r {t) 
Jo \Jo 

(3.50) 

roo / roo \ 

= EJ e- 7frW /i-Q r (t)df r (t)+Efy e-^^p-dU^t)), 

where the last equality follows from Lemma 3.10, with a = f r and c = r r 
[(3.49) is used for the first term and (3.48) with G = U r is used for the 
second term]. Recall the definitions of h and w(-) from Section 2. Define 

(3.51) f(x,y) = e-™h(x), xeM+,y£R + . 

Since h ■ q> h(w(q)) for all q € we have that the first term right-hand 
side of (3.50) is bounded from below by 

roo roc 

(3.52) E e-^ fr ^h{W r {t))df r {t)=E f{W r (t),f r (t))df r (t). 



Let (W{-),X(-),U(-),f(-),E(-), M'(-),M(-)) be as in Lemma 3.6 and let {r'} 
be the sequence along which convergence to this limit point occurs. Using 
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the Skorohod representation theorem (and relabeling the subsequence {r'} 
by {r}), we can assume without loss of generality that 

(W r (-),U r (-))^(W(-),U(-)) mV and 

(3.53) 

f r (-) — > f (•) in C a.s. as r — > oo. 

Define, for N > 1, /jv(-, •) = /(•, •) A AT. Using Lemma 3.11, we have for each 
fixed iV > 1 and u > 0, 



a.s. 



(3.54) / f N (W r (t),f r (t))df r (t)^ f N (W(t),f(t))df(t) a.s., 
jo jo 

(3.55) f U e-^ fr{t) p-dU r (t)^ r e -^p-dU(t) 
Jo Jo 

as r — > oo. Thus, a.s., 

liminf/ /(^(t),f(i))^(t) 
«/ 

/"U 

(3.56) >hminf/ f N (W r {t),f r (t))df r {t) 

J 



f N (W(t),f(t))df(t). 
Taking the limit as N — > oo and u — » oo in (3.56), we have that, a.s. 

/'OO /"OO 

(3.57) liminf / f(W r (t),f r (t))df r (t)> f(W(t),f(t))df(t). 

r ^°° Jo Jo 

Similarly, using (3.55), we obtain 

/* OO P OO 

(3.58) liminf/ e^^p ■ dU r (t) > e-~< f{t) p-dU(t). 

r ^°° Jo Jo 

Next, using (3.4), (3.50) and (3.52), we obtain 
J(q,{T r })= lim J r (q r ,r-) 

r — >oo 

roc 

= lim E / e- 7fr W/i(Q r (£))df r (t) 

r— oo J 

(3.59) + limE/ e -^ fr ^p-dU r (t) 

r^oo 7 
/•oo 

>liminfE/ /(T^ r (i),f r (i))c2f r (i) 
r ^°° Jo 

POO 

+ liminfE / e-^ r{t) p-dJJ r (t). 

rwoo J Q 
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Finally, using Fatou's lemma, (3.57), (3.58), (3.59) and (3.51), we have 



J(q,{T r })>E 



lim inf 

r— >oo 







+ lim inf 

1 >00 







(3.60) 



> E 
= E 
= E 



o 



f(W r (t),f r (t))df r (t) 
e-^ fr ^p-dll r (t) 
f(W(t),f(t))df(t) + 

oo 

f(W(t),t)dt + 

oo 

e-^h(W(t))dt + 



o 



[0,oo) 



e~ 7f {t) p ■ dU{t) 
e-^p ■ dU(t) 



[0,oo) 



7i p • dU{t) 



>Jo(w), 



where the equalities in the third line follow from Lemma 3.10 [with a = 
f, c = t in (3.49) for the first term and a = r, c = f , G = U in (3.48) for the 
second term]. The inequality in the last line of (3.60) is a consequence of 
Corollary 3.9. This completes the proof. □ 

4. Proof of Theorem 3.7. This section is devoted to the proof of The- 
orem 3.7. Since U r {-) is adapted to {.F{(i):t > 0}, it follows that for all 
s, a > 0, 

{f r (s) <a} = K(a) > s} = {a + £ > s) G ^(a). 

I m=l J 

This shows that for each s > 0, f r (s) is a {^"[(t) :t > 0} stopping time. For 
each r, t > 0, define the stopped sigma field 



(4.1) 



s r (*)=nr(*))- 



Since, for each r, {f r (t) :t > 0} is an increasing sequence of stopping times, 
it follows that {Q r {t) :t > 0} is a filtration. The following lemma gives an 
alternative representation of the filtration {Q r {t)}: 

Lemma 4.1. Define for fixed r and t>0, p r (t) = a r Q (f r (t)). Then p r (t) 
is a {J- r ((m,n))} stopping time, 

(4.2) g r (t)=T r (p r (t)) 
and U r {-) = U r { fr (-)) ^ {G r {t)}- adapted. 

Proof. Fix r, t > 0. Let {f£(t)} Q be a decreasing sequence of {F[ (s)} 
stopping times, such that for each a E N, f^(t) takes values in the countable 
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set T a = :j G No} and f£,(t) [ f r (t) as a — > oo. By right continuity of 
ctq(-), it follows that /9„(i) = Oo(f^(i)) J, //"(t), as a — > oo. 
Next, observe that for all (m,n) € N I+J , we can write 

(4.3) K(t)<(m,n)}= |J < {m,n)} n {f^(t) = s}. 

For each s G r Q , {f£(t) = s} G ^i(s) = ^(^(s)). Thus, by definition of the 
stopped sigma field ^(a^s)), the set in (4.3) is in T r ((m,n)) for all m,n, 
and so for each a, p r a (t) is a {.^((m, ra))} stopping time. Thus, p r (t) = 
inf a p^(t) is a {^"((m, n))} stopping time. 

To prove (4.2), define CT(t) = F"{p r {t)) and for each a, G r a (t) = J^(f^(t)) 
and Q r a (t) = T r (p r a {t)) for t > 0. Next, we show that for r,a,t > 0, 

(4-4) = 
Consider ie^(t). Then 

(4.5) An {p r a {t) = {m,n)}eJ rr ({m,n)) for all (m,n) G N I+J . 

Since f^(t) is a {^[(s) :s > 0} stopping time taking values in r Q , {f„(i) = 
s} g ^I(s) = r\a T Q {s)) for all s G T a . Hence, for all s G r a , (m,n) G N I+J , 

(4.6) {f r a (t) = s}n {a r (s) = (m,n)} G ^((m.n)). 

Taking the intersection of the two sets in (4.5) and (4.6), and recalling the 
definition of p r a {t), we obtain that for all s G T a , (m,n) G N I+J , 

(4.7) AH {f£(t) = *} n K(s) = (m,n)} G ^((m,n)). 

Thus, A n {f£(t) = s} G J=J(a) for all s G T Q , proving A G Q r a (t). This proves 
^a(i) C G T a (t). Next, we consider the reverse inclusion. If A G Q r a {t), then 
(4.7) holds, and taking the union over all s G T a , one gets (4.5), which 
implies that A G G r a {t). This completes the proof of (4.4). 

Note that if jk is a {J rr ((m,n))} stopping time with 7^ [ 7, then 7 
is a {.F r ((m, ra))} stopping time and .F r (7fc) |^ rr (7). Using this property 
and noting that o~q (i) is a right-continuous function of i, it follows that 
{J~i{t) = ^ 7r (cTg(t)) : t > 0} is a right-continuous filtration. This, together 
with (4.4) and the facts that f£(t) | f r (t) and p r a (t) j yields (4.2). 

The adaptedness of U r {-) to {G r (t)} is an immediate consequence of the 
adaptedness of U r to {F[(t)} and (4.1) (cf. Proposition 1.2.18 of [14]). □ 

The following lemma is needed for some estimates in the proof of Theorem 
3.7: 

Lemma 4.2. For each t G [0, 00) and p G N, 

sup max E((E[(t)) p ) < 00, sup max E((S^)) P ) < 00. 
r i=l,2,...,I r j=l,2,...,J 
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Proof. We will only prove the first inequality. Fix i G {1, 2, . . . ,1}. We 
claim that there exist e, 5 G (0, 1) such that 

(4.8) inf P«(l) > 5) > e. 

To prove the claim, we argue via contradiction. In view of Assumption 2.1, 
there exists an a € (0, oo) such that 

(4.9) inf E«(l)) > a. 

Let 5 = (a/4) A (1/2) and suppose that (4.8) fails to hold for this choice of 
5, for any e G (0, 1). Then there exists a sequence {rk]k>\ such that 

(4.10) P(u[ fc (l) ><5) < i. 

Ki 

This, in particular, implies that i < I' and that {w[ fc (l)K->i ^ s a tight se- 
quence. In fact, since sup r E(it[(l)) 2 < oo from Assumption 2.1, the sequence 
is uniformly integrable. Let u* be a limit point of this sequence along a 
weakly convergent subsequence. By uniform integrability and (4.9), we have 
E(it*) > a. Thus, P(u* > 25) > e for some e G (0, oo). However, in view of 
(4.10), and recalling that u* is a (weak) limit point of {u^ (1)} k>i , we see 
that P(u* > 5) = 0. Thus, we arrive at a contradiction. This proves (4.8). 
Now choose w G N such that w5 > t. Following the proof of Theorem 3.4.2 
of [9], we see that for all m G N and all r, P(-E[(i) > rmu) < (1 - e m ) m . The 
moment estimate in the statement of the lemma now follows. □ 

Martingales associated with M r {-). Define for m G N , n G N J , i' = 1, . . . ,1', 
J = !,■•■, J, i = !,...,!, 



J * " * 5 3 



(4.11) 



ii'(m,n) =|[/(mj/) = - ^(1 - a-,u[(n)), rrty > 1, 
r n=l 

<&((m,n)) = ^K) = I EC 1 ^ ^ L 

n=l 



fc=i 

As a convention, we take £[((m,n)) = for i = I' + 1, . . . ,1. It is easy 
to check that for all i = 1,...,I, j = 1,...,J, ££(•), ^J(') and Cf(-) are 
{.F r ((m,ra)) :m G N 1 ,?! G N J } martingales. Straightforward calculations give 
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the following quadratic variations for these martingales. For ^ %2 € 

{l,... l l},j,ji^ 2 e{l,...,J} J 



I r u , r \2 



(Vj)({ m , n )) 
(CD((m,n)) 



(4.12) 



{4,&)(Kn)) = (f[,^)((m,n)) = 0, 

(^,^ 2 )((m,n)) = (^,C;)((m,n)) = <£,&)((m,n)) = 0, 

(c; ; c 2 )(K-)) = -^ =inj( ^ 2) 



Let p r (t) be as in Lemma 4.1. Using the fact that /9 r (t) is a {.F r ((m, ra))} 
(multiparameter) stopping time, we have that 

N r (t) = (N?' r (t),i = l,...,I; 

N?> r (t),j = l,...,J;N*> r (t),i = l,...,I) 

(4-13) 

= (W(t)),i = i,...,i; 

^(p r (t)),j = l,...,J;CKp r (t)),i = l,...,I) 

is a {£/ r (i) :t > 0} martingale. This multiparameter version of optional sam- 
pling theorem can be proven in a manner similar to the corresponding sin- 
gle parameter case (cf. Theorems 4.7.4 and 4.7.5 of [9]). The key conditions 
needed to invoke the theorem are the following. For all i = 1, . . . , I, j = 
1 , . . . , J , m i ,rij = 1,2,..., 

n\p r (t)\] <oo, 

EM( mi + 1) - [[(^(^(Kn))] < c r , 

(4.14) 

E[\^(n J + l)-^(n,)\\r((m,n))]<c r , 

KWCrin, + 1) - Cnn^Wr {(m,n))] < c r , 

where c r 6 (0, oo) does not depend on (m, n). The last three bounds in (4.14) 
follow from the definition of the martingales and the fact that for fixed r, 
i and j, {u^(n), Vj(n), ^' r (n)} is a sequence of i.i.d. random variables with 
finite variance. For the first condition in (4.14), note that using f r {t) < t, 
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Tj(t) < t for all t > 0, we have, for some k 6 (0, oo), 

E[|pr(t)|]<« 



3-3 



£E(£[(rV (t) + 1)) +^E(5}(7j(r 2 f(t)) + 1)) 
i=i j=i 



(4.15) 



< K 



Y,HEl{{r 2 t) + 1)) + ^E(S;((r 2 () + 1)) 
i=i j=i 



< oo. 



This proves (4.14) and hence the martingale property claimed below (4.13). 

Using (4.12) and the above martingale property, it is easy to verify that 
the following hold for any t\ > t\ and all i,*i,*2) £ {!> • • • > I}j J £{!>••• > J} : 

E[(j«f' r (^)-j«f' p W)) 2 isrw)] 



(4.16) 
(4.17) 
(4.18) 



= (^ r ) a E[(^(f r (^))-^(f r («J)))|(T(*I)], 

= (/3K' r ) 2 E[(^(7T(T r (tD)) - ^(rj(f(t r ))))|^^)], 



=E(^)E[(^(^(f'-(^)))-^(27(^(ti))))|<r(t;)]. 

3=1 

Furthermore, for / i 2 e {1, . . . ,1}, ji / j 2 , £{!,-■ • , J}, 

E[(^7«)-^f ,r W))(^f r (^)-^ ,r W))l«rW)] 



(4.19) 



E[(iV^(f 2 ) - ft£' r &))(ti% r (&) - N^wwm 
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= E[(iV*' r (iD - ^• r (*I))(^' r (^) - N^itDWitl)] 

= E[(tf*' r (tS) - 2^' r (tJ))(#f - ivf • r (t5))|(T(^)] 
= 0. 

Relationship between M r (-) and N r (-). Recall the definitions of M r and 
N r given in (3.36) [see also (3.17)] and (4.13), respectively. We now show that 
the difference between M r (-) and N r (-) approaches as r increases. Note 
that for t = I' + l,...,I, M l E ' r (-)=Nf' r (■) = (). Now consider i € {1,...,I'}. 
It is easy to establish that for all r and t > 0, 

(4.20) Elit) + ^ 0>1 < &{oftt)) < + -• 
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Recalling f r {t) < t for all t > 0, we see that 
sup \M l E ' r (s)-N l E ' r (s)\ 

0<s<t 

(4.21) 

1 r [suPo< s <tM(El-(r 2 s) + !)-!/<)] 

< - + a- . 

r r 

Observing that (fc) — l/a£ : = 1, 2, . . .} is a sequence of i.i.d. zero mean 
random variables, we see that max fc <( a . t+1 ) r 2 \ul(k) — \ja\\jr — > in prob- 
ability, for all fixed t > 0. Using this observation, together with the fact 
that 

(4.22) iV ' ai e(-) asr^oo, 

it follows that 

[sup 0<s< ,(<(£f(r 2 s) + l) -!/<)] 

(4t.zoj — ► u as r — ^ oo, 

r 

in probability. Hence, using (4.21) and (4.23), we have for all i = 1, . . . ,1, 
(4.24) |jwrf-»-(.)_Jvf' r (.)H0 asr^oo, 

in probability u.o.c. 

Similar arguments, using the observations that TJ(s) < s for all s > and 

4'i' r i n ij) i Pi — 1 f° r an n «i — 1) §i ve the following two bounds: 
sup \M?> r (s)-N* r (s)\ 

0<s<t 

(425) <1 , ^ [supo^^^-^ + l)-!/^)] 



sup |i#' r W-Af' r ( a )|<^. 



One can now argue, as before, that for j = 1, . . . , J, i = 1, . . . , I, 

(4.26) \M?' r (-)-N?> r (-)\^0, |M^(.)_iv^(.)|^o asr^oo, 
in probability u.o.c. Thus, we have shown that 

(4.27) \M r (-) - N r {-) \ ^ in probability u.o.c. as r -> oo. 

PROOF of Theorem 3.7. The first part of the theorem is immediate 
from Lemma 3.6(C) and the fact that f (r(i)) =t, t > 0. Using (2.21)-(2.24) 



CONTROLLED STOCHASTIC NETWORKS 35 
and (3.18), we get, for all t > 0, 

(4.28) X r (t) = M E ' r (t) + (C - P')M S ' r (t) + M*' r (t), 

(4.29) Y r {t) =r(x*t-T r (t)), U r (t) = KY r (t) , 
W r (t) = Aq r + A[0Jt - (C - P') diag(^)T r (t)] 

(4.30) 

+ Al r (t) + GL> r (t). 
Replacing t by r r (i) in the above, we get 
(4.31) X r (t) = M E ' r (t) + {C- P')M S ' r (t) + Af*' r (t), 

W r (t) = Aq r + A[6lf r (t) - (C - P') diag(^)f r (f r (t))} 

(4.32) 

+ AX r (t) + GU r {t). 
On passing to the limit in (4.31), (4.32) and using (3.42), we have 
X{t) = M E (t) + {C- P')M s (t) + M®(t), 

(4.33) 

W{t) = Aq + A9t + AX(t) + GU(t), 

for all t > 0, where is as in (2.39). 
Define the sigma fields 

H'{t) = a{M(s), U{s),f(s) : a < t}, 

(4.34) 

H(t)= p| n'f t+ ±), t>o. 

n>l 



By construction, {"H(t)} is a right-continuous filtration, and for all u > 0, t > 
0, the following holds: 

(4.35) {r(t) <u} = {f(u)>t} eH(u). 

Using the right continuity of {H(t)}, we now have that for all t > 0, r(t) is 
a {7Y(i) } stopping time. Define the stopped sigma fields 

(4.36) H(s)=n(r(s)), s>0. 

Since {r(i) : t > 0} is a nondecreasing sequence of stopping times, {TC(t) : t > 
0} is a filtration. Furthermore, the processes W,X and U are all linear 
combinations of the components of M and U and are hence adapted to 
{H{t):t> 0}. This shows that W, X and U are adapted to {H(t):t> 0}. 
We will now establish that A is a (0, E)-Brownian motion with respect to 
this filtration, thereby completing the proof of Theorem 3.7. 

Let N = I + J + 1 = 21 + J and / G C c °° (R N ) = space of infinitely differen- 
tiable functions from M. N to R with compact support. Let x = (x s , x 5 , x*) = 
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(xf, i = 1, . . . ,1; Xj,j = 1, . . . , J; xf,i = 1, . . . , I) € M. N . Define the second or- 
der differential operator G as follows: 

Gf(x) = Gf{x E ,x s 7 x*) 



(4.37) 



2 a H fx E x E ( X ) 2 E/ (J jjfx s x? ( X ) 



1=1 



i=i 



+ 5 E E a ?iiJx*x* 0*0, 



tl=l 42 = 1 



where for i = 1, . . . , I, j = 1, . . . , J, ii,i2 = l 



7 * ' * 5 J 



(4.38) af i =on{ai&, 



u\2 
i I i 



a 



j. i 



PjxUfra]) 2 , 



a 



3=1 



In order to complete the proof, it suffices to show that for any t, s > 0, 

rt+S 



(4.39) E 



f(M(t + s))-f(M(t)) 



Gf(M(u))du\H(t) 



0. 



We claim that, in order to prove (4.39), it is enough to prove the following: 



E 
(4.40) 



h(M(s k ),U(s k ) :s k <t,k = l,...,n)( f(M(t + s)) - f{M{t)) 

t+S Gf(M(u))df(u) 



0, 



where h is a bounded continuous function of its arguments. In order to justify 
the claim, define Y f (t) = /(M(i)) - /„ Gf(M(u)) df(u), t > 0. Then (4.40) 
implies that {Yf(t)}t>o is a {H'(t)}t>o martingale and hence a {7i(t)}t>o 
martingale. We will now use optional sampling theorem to show that 
{Yf(,T(t))}t>o is a {7~L(t)}t>o martingale. Using the fact that both / and 
Gf are bounded (by some constant c > 0) and (3.37), we have, for all t > 0, 



n\Yf(r(t))\] 



<E 



f(M(r(t))) 



[0,T(t)] 



Gf(M{u))df(u) 



(4.41) 



<c(l + i), 
E[|*KT)|I {tW >t}] 



<E 



f(M(T)) 



Gf(M{u))df(u) 



'[0,T] 

<cE((l + i)I {f(T) < t} ) 

< c(l + t)P[f(T) < t] -»■ asT^oo, 



L{f(T)<t} 
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where the last convergence is a consequence of Lemma 3.6(B). Hence, it 
follows from Theorem 2.2.13 of [10], on observing that r(t) is an a.s. finite 
{Ti.(s) : s > 0} stopping time for each t > 0, that {Yf(r(t))}t>o is a {Tt(t)}t>o 
martingale. Thus, using Lemma 3.10, (4.39) follows. This proves the claim. 

We now show (4.40). Fix t,s > 0. Let f m =t + ^s, m = 0, 1, . . . ,r 2 - 1. 
Define 

I r 2 -l 

* r =^E E /Afr(C))(«kr r ) 2 (^(^(c + i))-^(^(C))) 
+^E r E/xf^(^ r (c))(/3^;' r ) 2 

j'=l m=0 

(4.42) x (£;er^(c + i))) - 5j(rj(f r (C)))) 

I I r 2 -l 



'1 '2 

zi =1 ^2=1 m=0 

x E(< 2 )(^ r (^ r (^(c + i))) - 5j(ij(f r (c)))). 

Fix the subsequence along which the convergence to the limit point in 
Lemma 3.6(B) takes place and, by relabeling, denote it again by {r}. In 
view of (4.27), we have, in particular, that as r — > oo, 

(4.43) (M r ,N r ,f r ,U r ) (M,M,f,U). 

By appealing to the Skorohod representation theorem, assume without loss 
of generality that the above convergence holds a.s. 
We will now show the following two results: 



(4.44) limsupE 



r— >oo 



t+s 



f r - / Gf(M(u))df(u) 



0. 



limsupK[h(N r (s k ),U r (s k ) :s k <t,k = l,...,n) 

(4.45) 

x(f(N r (t + s))-f(N r (t))-^)] = 0. 

Note that since both h and / are bounded and continuous, on applying the 
continuous mapping theorem and dominated convergence theorem, one gets 
(4.40) from (4.43) and (4.44)-(4.45). Thus the proof of the theorem will be 
complete once (4.44) and (4.45) are established. 
We first consider (4.44). For u £ [0, s), define 



r 2 -l 



iv['*H=^iv r (C)i [w+i) (i + w ). 

m=0 
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Prom (4.43), it follows that asr^oo, iV['*(-) -> M(t + ■) in T>([0,s)), a.s. 
Also, combining (4.43) with Lemma 3.6(C), we see that E^(f T (-)) — > ct^f (•) 
in P. Hence, using Lemma 3.11, we have the following result for the first 
term on the right side of (4.42): 



I r 2 -l 



(4.46) ±£ E /^(^ r (C))(«r^* r ) 2 (4 r (r r (C + i))-A r (r r (C))) 



i=l m=0 



(4.47) 



IE 

i=i 

►IE 

i=l 
t+s 



t+s 



t+s 



f x E x E{M(u)){a,iG™) otidf(u) 



* * % % 

i=l 



almost surely. Using the bound K[(E^ (r 2 t) / r 2 ) 2 ] < c(t 2 + 1) for the renewal 
process E\, where c is a constant independent of r and t, it follows that the 
expected value of the square of the term in (4.46) is bounded by 

KR[{El(f r (t + s))) 2 } < Kc((t + s) 2 + 1), 

for some n > 0. This estimate gives us the required uniform integr ability 
enabling us to conclude that 



E 



I r 2 -l 

EE. 

i=l m=0 



(4.48) 



IEEW(^ r (C))(«>D 2 

x(Ei{r\f m+l ))-Ei{r\f m ))) 



t+s 1 

2 



i=l 



0. 



Using similar arguments for the other two terms in ^ r in (4.42) and the 
definition of the operator G in (4.1), one gets (4.44). 

Finally, we consider (4.45). Recall that U r (-) and N r {-) are adapted to 
{Q r (t)} [see Lemma 4.1 and remarks below (4.13)]. Thus, h(N r (sk),U r (sk) '■ 
Sfc < t, k = 1, . . . ,n) is measurable w.r.t. Q r {t). Hence, by boundedness of h 
and / and uniform integrability of \& r established above, in order to prove 
(4.45), it is enough to establish that 



(4.49) limsup\E[f(N r (t + s))- f(N r (t))-V r \g r (t)}\ = a.s. 
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To prove (4.49), we first rewrite the difference f{N r {t + s)) — f{N r {t)) as a 
telescoping series and expand each term using Taylor's formula: 

E[f(N r (t + s))-f(N r (t))-* r \g r (t)} 



E 



(4.50) 



E (/ (^ r (C+i)) - f(N r {f m ))) - «r|*T(t) 



_m=0 



N 



E E 



U'=l Lm=0 



H-l 



E ( WC) - w^ r (C))) 



x (TVf (i 



r 1 
m+1; 



m+1) 



m r m )W(t) 



where N r (-) = (N[(-) : I = 1, . . . , N) and lies in the line segment joining 
N r {t r m ) and N r (t r m+l ). In obtaining the last equality, we have used the 
fact that {N r (t)}t>o is a {£/ r (i)}t>o martingale [see below (4.13)], and also 
applied (4.16)-(4.19). Define, for c> 0, 

(4.51) B r c ' m = {\r(N r (f m+1 ) - N r (f m ))\ > c}. 
We claim that for each e > 0, there exists a c > such that 

(4.52) sup sup E[\r(N r (f m+1 ) - N r (f m ))\%, m \g r (t)] < e. 

r m=0,...,r 2 — 1 

Postponing the proof of (4.52), we now complete the proof of (4.49), assum- 
ing (4.52) holds. Note that for 6 r m as in (4.50), we have 

(4.53) 1^-^(01 < K(C+i) -N r (t r J\. 
Also, clearly, for all I, I' = 1, . . . , N, 

\(N[(t r m+1 ) -N[(t r m ))(NHf m+1 ) -NWm))\ 
<\N r (f m+1 )-N r (t r J\ 2 . 



(4.54) 



Since f Xl x l ,{') is bounded (by qj/, say), we have, using (4.52), (4.53) and 
(4.54) in (4.50), 

\E[(f(N r (t + s))-f(N r (t))-^W(t)]\ 

N r 2 -l 

<2 E E c M^[|iV r (C + i)-^ r (C)l%-I^W] 

l,l'=l m=0 
N r 2 -l r 



(4.55) 



+ E E 

Ll>=l m=0 



sup \f Xl x l ,{x)-f XlXl Xy)\ 

\x—y\<c/r 
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|2i 



xE[\N r (f m+l )-N r (C)\% B r, mr \gr {t )] 



/ N \ N r 2 -l 

\Ll'=l ) U'=lm=0 



SUP 1/^,0) -fxix,,(y)\ 
\x—y\<c/r 



Taking lim sup as r — > oo in (4.55), recalling that / G C™(R N ) and that e > 
is arbitrary, the statement in (4.49) follows. 

Finally, we prove the claim made in (4.52). Note that it is enough to 
prove that the sequence {\r(N{(t'^ n+1 ) — ^VJ(*m))| 2 } i s uniformly integrable 
[conditioned on Q r (t)] for each I = 1,...,N. We will only check this for 
i = 1, . . . ,1 (i.e., I = 1, . . . ,1) here: the proof for other values of I (i.e., I = 
I + 1, . . . , 21 + J = N) is similar and is omitted. Thus, we will show that for 
i = 1, . . . , I, given any e > 0, there exists c > such that 

sup sup E[|r(iVf > r (t r m+1 ) -N?> r (fJ)\ 2 

r m=0,...,r 2 -l 

(4.56) 

X I {|r(JVf' r (^ +1 )-JVf'*-(^))|>c}l^ T (*)] <£ a ' S - 

From the definitions of N E ' r {-) and ££(•) [see (4.13) and (4.11)], we get that 
for i = 1, . . . , I' [the case i = I' + 1, . . . , I, (4.56) holds trivially], 



^f' r (C + i)-^f' r (4)) 



(4.57) =r(£( P nC +1 )) 

= (p? £ (c+i) - p? £ (0) - aT«w*(CH)) - er(p? E (C)))- 

Using the inequality f r (i + s) — f r (i) < s for all s,i > and recalling the 
relation between E\ and £!■ [see (2.5)], it follows that 

IP? - P? E (t r m )\ < ^[(rV(C) + *) - El(r 2 f r (f m )) 

(4.58) 

< mm E ifm)) +s)- Ei{^ E {f m ))) + 1. 

Observing that p V i E {t r m ) is a {.F r ((TO, n))} stopping time and using proper- 
ties of renewal processes, we obtain 

(4.59) 



p[*T(W ,J5 (0) + *) - ^(er(p? E (C))) e -is r wi 



<p[Kr( S )+ie-], 

d 

where for probability measures fj,, v on R, we write /i<z/ if /i(— oo,x] > 
oo,x] for all x € K. Proof of (4.59), which is left to the reader, can 
be given along the lines of [22] (page 63). This, in particular, implies the 
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following moment estimate for the term on the left-hand side of (4.58): 

H(p r i E (t r m + l) ~ pl' E (fJ) 2 I {]p r, E{tr )-^ te )|> c} |S r (*)) 

(4.60) 

<E((E^(s) + 2) 2 I {E r {s)+2>c} ). 

Using Lemma 4.2, we now have that the supremum (over r) of the right- 
hand side of the above inequality converges to zero as c — ► oo. This proves 
the (conditional) uniform integrability of the sequence corresponding to the 
square of the first term in (4.57). Thus, it suffices to prove that the square 
of the second term in (4.57) is (conditionally) uniformly integrable. For 
this, note that if p^' (t r mJrl ) > pl' E \t T m ), then using the fact that f r (t r m+ i) — 
f r (t r m ) < (t + (m + s)/r 2 ) - (t + ms/r 2 ) < s/r 2 , it follows that 

UP? E (t r m+ i)) ~ Up? E (t r m )) < r 2 (r r (C+i) " ? r (0) + <(P? E (C+l)) 
(4.61) 

<s + ul(pf(f m+1 )). 

For the remaining case, that is, if p^ (t r m+ \) = p r { E iP~ m )^ the difference be- 
tween the two terms in the left-hand side of (4.61) is zero. Hence, we have 

fe(p? B (C+i)) - ^(p? £ (C))] 2 i { | 6(p ^ (c+i)) _ fi(p ^ ( ^ )) | >c} 

(4.62) < 2[s 2 + K(P? £ (C + i))) 2 ]I K(p ^ (c+i))>c _ s} 

<m(p? E (t r m+1 ))-^ E (f m m 

where ££(m) = Y,T=i[s 2 + <(*0 2 ]I{«T(fc)>c-s} for m > 1. Note that {[s 2 + 
^(fe) 2 ]I{ n r( fc ^ >c _ s |} fc > 1 is a sequence of i.i.d. random variables with finite 

mean and £f((m, n)) = Ci( m i) i s an F r ((m, re)) martingale. Since Pi' E {t T m+ i) > 
Pi ,E (trn) are * wo ^((m, n)) stopping times, using the optional sampling the- 
orem, one gets the following from (4.62), 

(4.63) < 2E(p^(C +1 ) - p? E (4)|<7(4))E[( S 2 + <(l) 2 )I K(1)>c _ s} ] 

< 2E(E r l (s) + 2)E[( S 2 + <(1) 2 )I K(1 )> C _ S} ], 

where the second inequality follows on using (4.58) and (4.59). Finally, since 
{u[(l) 2 } r is uniformly integrable [see (2.2)], and the expected value of El(s) 
is bounded in r (see Lemma 4.2), the right-hand side above goes to zero as 
c —¥ oo. Thus, the square of the second term in (4.57) is (conditionally) 
uniformly integrable. This proves (4.56), and hence the proof of Theorem 
3.7 is complete. □ 
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5. Admissibility of control policies. In this section, we will show that for 
a very broad class of sequences of control policies {T r } which satisfy some 
natural nonanticipativity conditions, the admissibility requirement (iv) of 
Definition 2.6 is satisfied. Throughout this section, we will fix r and consider 
the rth network N r . Consider a control policy T r which satisfies (i) , (ii) and 
(hi) of Definition 2.6. We will show that under further natural conditions 
on T r , the property (iv) of Definition 2.6 also holds. For the rest of the 
section, we will suppress r from the notation, unless it is necessary. The first 
condition on the policy T is the following: 

Assumption 5.1. There is a J-dimensional measurable process T with 
values in {0, 1} such that T{t) = JqT(s) ds for all t G [0, oo). 

The second condition on the policy states that the process T does not 
change values between two successive event times, where an event is either 
an exogenous arrival into the system or the completion of a service by some 
server. To state this condition precisely, define To = 0, and for any £ > 0, let 
Ti + i be the first time after T^ when either an arrival or a service completion 
takes place. From (2.1), it follows that, almost surely, {' ) Ce}eeN is a strictly 
increasing sequence, increasing to oo as £ — > oo. Here we assume that multiple 
events can occur at a given T^. 

Assumption 5.2. For all £ G N , f(t) = f{T e ) for all t G [T e ,T e+1 ). 

Our final condition on the policy is a natural nonanticipativity property. 
In order to state this condition, we introduce the following notation. For 
i G {1, . . . , 1} and £ G No, let u\ = ^(^(T^) + 1) — T^. Thus, u\ is the residual 
(exogenous) arrival time at the ith buffer at time T^, unless an arrival of the 
ith class occurred at time Tg, in which case it equals u\ = Ui(Ei(Ye) + 1). 
Similarly, for j G {1, . . ., J}, I G N , define v] = rjj(Sj(T e ) + 1) - T e . Next, 
for i G {1, . . . , I}, set Q^o = 0> an d for £ > 1, Qin = Qi(Ti). Also, for j G 
{1, . . . , J} and £ > 0, let if = fj(T e ). Also, let Tj 1 = 0. Finally, define, for 
£>0, 

(5.1) ^ = {(T^,tif,«f J Qf,3f- 1 :iGl,... > I,iel,...,J):^ = 0,...,£}. 

Assumption 5.3. For all £ > 0, T(T e ) is a measurable function of x £ - 
The following is the main result of this section: 

Theorem 5.4. LetT be a control policy for J\f r that, in addition to (i), 
(ii) and (iii) of Definition 2.6, satisfies Assumptions 5.1, 5.2 and 5.3. Then 
T satisfies condition (iv) of Definition 2.6. 
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Proof. We begin with some notation. For a given activity j G {1, . . . , J}, 
let denote the associated buffer. For a given buffer i G {1, . . . ,1}, let Ji 
denote the set of all activities that are associated with the buffer i. Define, 
for each J C J, L and a vector a G {0, 1} J , 

(5.2) n(a,Ji)=( U aA ( JJ (1-aA 

Note that for JiQJi, ir(a,Ji) is either zero or one, and it takes the value 
one only if a,j = 1 for all j G Ji and aj = for all j G Ji\Ji- Denote this 
unique J by Ji{a). A straightforward calculation shows that 

(5.3) T m = T e + min min{uf , : j G J(f e )}. 

ie{i,...,i} 

Also for i = 1,...,I, let be the indicator function that at T^+i, the 

event (arrival or service completion) occurred at buffer z. More precisely, for 
i = l,...,I and^>0, 

'l, if minK,^:jG Ji(T £ )} 

(5.4) Zf +1 = < = min min{u^,,uj : j G Ji'(T^)}, 

1 0, otherwise. 
From (5.4), (5.3) and Assumption 5.3, it follows that both 

(5.5) lf +l ,Ti + i are measurable functions of % ■ 
For I > and (m, n) G N I+J , let 

iB^ n = {^(T / ) + l = m i , 

(5.6) 

SjiTjCTz)) + l = nj-.i = l,...,l, j = 1, . . . , J}. 
We claim that the following two statements hold for all I = 0, 1, . . . : 

(A) B e mn e JF((m,n)) for all (m,ra) GN I+J , 

(5.7) 

(B) Ig^ n / G ^"((m, n)) for all (m, n) G N I+J , 

where lc denotes the indicator function of a set C. We will show (5.7) by 
induction on £. Note that for I = 0, B IL mn is the sample space if (m,n) = 1, 
otherwise it is the empty set. Thus, for £ = 0, (A) holds trivially. Also, when 
(m,n) = 1, one sees from (5.1) that x° £ J r ((m,n)). So (B) also holds for 
the case £ = 0. Now suppose that (5.7) holds for some t G No, and consider 
(5.7) with £ replaced by t + 1. 

Note that the set i?^"^ can be represented as 

(5-8) a$g= (J C n Cn- 

(p,g)<(m,n) 
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We will next show that 

(5.9) B^ q n B*£* G F((m, n)) for all p < m, q<n. 

This, in view of (5.8), will prove that (A) in (5.7) holds with I replaced by 
£+ 1. By induction, Bi„ G J-((p,q)) C .^((m, n.)) for all p < m, g < n. Also, 
on Sp )(J , the following representations hold. For i G {1, . . . , I}, j G {1, . . . , J}, 

(5.10) [Ei(T i+1 ) + 1] = ft +Xf +1 I K< ^ : jeJi{ f* )} , 

(5.11) [SjfflM + 1] = % + W^<^^:i^ w ^W^}- 

Noting that the right-hand sides of (5.10) and (5.11) are measurable func- 
tional of we see from (5.6) that 

(5-12) I e e+1 = I Bt y £ ( X e ) 

for some measurable function Now by induction, the right-hand side of 

(5.12) is in T((p,q)) and therefore in J-((m,n)). This proves (5.9) and hence 
(A) in (5.7) holds with I replaced by £ + 1. 

Finally, we verify (B) in (5.7) for £+1. Note that on B*+£, for i = 1, . . . ,1, 

(5.13) Qf 1 = <f + (m, - 1) - (nj - 1) £ Cy + £ " !)■ 

Thus, using (5.7)(A) for £+ 1, we see that <5f +1 I„£+i G T{(m,n)). Recalling 

that and are measurable functions of x^i we see from (5.8), the 

induction hypothesis and (5.7)(A) for £ + 1 that Ig«+i (T^ + i,T ) G ^({ra^n)). 
Hence, we have shown 

(5.14) I Bl+1 (T m ,Qf G {1,. . . ,1}, j G {1,. . . , J}) G ^((m,n)). 
Next, on B^\, for i G {1, . . . , I}, j G {1, . . . , J}, we have 

(5.15) = I{ u f>T, +1 -T f }04 " T ^+l + + r {uf=T, +1 -T,}^( m i)' 

= (1 " + f}(l - T%]){v* - T i+1 + T<) 

(5.16) + Tffi^I^e^ :fej l{j) (f% j '^} v j( n j) 

The above two identities, together with (5.8), (5.5) and the fact that (5.7)(A) 
holds with £ + 1, show that (?4 +1 , v j +1 ) G .^((m, ra)). Combining this 

with (5.14) and using the induction hypothesis [along with (5.7)(A) for £+1], 
we now have that (5.7)(B) holds for £ + 1. This proves (5.7) for all I G No- 
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Finally, using (5.5) and (5.7), we have, for each fixed t > and all (m, n) G 
N I+J , 

B m ,n = {E % {t) + 1 = m^SjiTjit)) + 1 = nj : i = 1, . . . = 1, . . . , J} 

(5.17) 

= U S ™,n n {T £ < t < T £+1 } G :F((m, n)), 

£=1 

where p = 2i=i m « + 2j=i n i- This proves the first part of condition (iv) in 
Definition 2.6. To show the final part of this condition, it suffices to prove 
that for < tj < t, j = 1, . . . , J, and all (m, n) G N I+J , 

(5.18) {Tj(t) < tj, j = 1, . . . , J} n B m>n G T((m, n)). 

From the properties of T [Assumptions (5.1), (5.2)], we have 

{T j (t)<t j ,j = l,...,3}nB m , n 

= (jB^ n n{T e <t<T e+1 } 

(5.19) 

n {Tj(? e ) + Tj(T £ ) (t - T £ ) < t 3 ; j = 1, . . . , J} 
G J-((m, n)), 

where the last inclusion follows on using (5.7) and (5.5). This proves (5.18) 
and, hence, the theorem. □ 
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